It seems like in some cases Xcode 12.3 launch way to many swift compilers and starts swapping heavily. Check for memory pressure in Activity Monitor. If it's red, try to fall back to Xcode 12.2 (helped me).
Topic:
App & System Services
SubTopic:
Core OS
Tags: